experiment: add reasoning monitor agent for prompt injection detection - #259
Closed
waynesun09 wants to merge 5 commits into
Closed
experiment: add reasoning monitor agent for prompt injection detection#259waynesun09 wants to merge 5 commits into
waynesun09 wants to merge 5 commits into
Conversation
…njection detection Proof-of-concept for issue #174. Three complementary post-execution monitors: canary token hook, tool allowlist hook, and LLM reasoning monitor that evaluates stripped transcripts blind to user inputs. Includes 15 attack payloads, synthetic transcripts, and test suite. Signed-off-by: Wayne Sun <gsun@redhat.com>
… attacks, and expanded test coverage
Review-driven improvements to the reasoning monitor experiment:
- Rewrite all 16 compromised transcripts to use realistic agent reasoning
that rationalizes malicious actions through normal triage logic, instead
of cartoonish self-aware confessions ("I found the hidden instructions")
- Add 5 benign control payloads (security bug, multi-label, cross-reference,
dependency update, docs update) for meaningful false positive measurement
- Add 5 Phase 2 adaptive attack payloads targeting the monitor's hardest
cases: normal-reasoning mimicry, allowlist-only attacks, transcript
truncation exploitation, cross-team escalation mimicry, and tool-argument
exfiltration
- Fix strip_user_input to redact ALL user messages, not just the first one
- Add mcp__github__search_issues to USER_CONTENT_TOOLS to prevent
second-order injection via search results
- Fix indirect-external-link and indirect-multistep payloads to remove
inline test scaffolding ([Attacker controls that URL...])
- Fix README table to list all 27 payloads with correct counts
- Add integration tests running deterministic monitors against all payloads
- Add Phase 2 test suite validating adaptive attack structure and monitor
gap analysis
Signed-off-by: Wayne Sun <gsun@redhat.com>
…fix README wording - adaptive-truncation-exploit.jsonl line 52: replace "planted unauthorized reporting task" with plausible triage reasoning - README.md: update "initial user message" to "all user message content" to match the updated strip_user_input behavior Signed-off-by: Wayne Sun <gsun@redhat.com>
Site previewPreview: https://5b2abde5-site.fullsend-ai.workers.dev Commit: |
- Split truncation (first 8K + last 8K) to catch late-stage compromises - Add Bash to USER_CONTENT_TOOLS to prevent second-order injection via cached file reads - Add user messages to all 27 transcripts so strip_user_input() user redaction path is exercised in integration tests - Add sliding-window rate limiting to LLMMonitor (env-configurable via FULLSEND_MONITOR_MAX_CALLS and FULLSEND_MONITOR_WINDOW_SEC) - Use pre-serialized error responses in hooks to guarantee output on failure even if json.dump itself fails - Convert Monitor base class to abc.ABC with @AbstractMethod - Remove broken cross-repo link in README - Consolidate duplicate dev dependencies in pyproject.toml Signed-off-by: Wayne Sun <gsun@redhat.com>
- Fix E501 line-length violations in MONITOR_SYSTEM_PROMPT and user_message - Fix E731 lambda assignment in format_results_table (use named def) - Fix E402 misplaced import in test_hooks.py (move Path to top) - Fix B905 zip() without strict= in test_monitor.py - Fix B007 unused loop variable in test_monitor.py - Add nosec B105 for intentionally hardcoded test canary tokens Signed-off-by: Wayne Sun <gsun@redhat.com>
Member
|
Hello! We are moving experiments to another repo because we are bloating this one, please reopen this PR at https://github.com/fullsend-ai/experiments. Thanks |
Member
Author
|
Ported to the experiments repo: fullsend-ai/experiments#2 Closing this PR since experiments now live in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
uv run pytest tests/ -x -vpasses all unit and integration testsuv run python runner.py --dry-runcompletes without errorsuv run python runner.py --monitor canarydetects canary leaks in expected payloadsuv run python runner.py --monitor allowlistblocks out-of-role tool callsuv run python runner.py --monitor llm --model haikuproduces verdicts for all 27 payloadsCloses #174